Learning a set of directions

〈μ,D〉 = ∑ i=1 wi 〈ui,uiu T i 〉 Gradient descent Mill maintains the two moments 〈μt,Dt〉 ∈ U as parameter At trial t = 1 . . . T, the Mill 1. Decomposes parameter 〈μt,Dt〉 into a mixture of directions and draws ut from mixture 2. Receives Wind direction xt and gain E [ (ut xt + c) 2] 3. Updates 〈μt,Dt〉 to 〈μ̂t+1, D̂t+1〉 with the gradient of the expected gain on xt μ̂t+1 := μt + 2ηc xt and D̂t+1 := Dt + η xtx T t 4. Projects 〈μ̂t+1, D̂t+1〉 back into U 〈μt+1,Dt+1〉 := argmin tr(D) = 1 D μμT ‖D− D̂t+1‖F + ‖μ− μ̂t+1‖